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Summary 

Over the past decade postsecondary education has been moving increasingly from the class¬ 
room to online. During the fall 2010 term 31 percent of U.S. college students took at least one 
online course. The primary reasons for the growth of e-learning in the nation’s colleges and 
universities include the desire of those institutions to generate new revenue streams, improve 
access, and offer students greater scheduling flexibility. Yet the growth of e-learning has been 
accompanied by a continuing debate about its effectiveness and by the recognition that a num¬ 
ber of barriers impede its widespread adoption in higher education. 

Through an extensive research review, Bradford Bell and Jessica Federman examine three 
key issues in the growing use of e-learning in postsecondary education. The first is whether 
e-learning is as effective as other delivery methods. The debate about the effectiveness of 
e-learning, the authors say, has been framed in terms of how it compares with other means 
of delivering instruction, most often traditional instructor-led classroom instruction. Bell and 
Federman review a number of meta-analyses and other studies that, taken together, show that 
e-learning produces outcomes equivalent to other delivery media when instructional conditions 
are held constant. The second issue is what particular features of e-learning influence its effec¬ 
tiveness. Here the authors move beyond the “does it work” question to examine how different 
instructional features and supports, such as immersion and interactivity, influence the effec¬ 
tiveness of e-learning programs. They review research that shows how these features can be 
configured to create e-learning programs that help different types of learners acquire different 
types of knowledge. In addressing the third issue—the barriers to the adoption of e-learning in 
postsecondary education—Bell and Federman discuss how concerns about fraud and cheating, 
uncertainties about the cost of e-learning, and the unique challenges faced by low-income and 
disadvantaged students have the potential to undermine the adoption of e-learning instruction. 

Based on their research review, the authors conclude that e-learning can be an effective means 
of delivering postsecondary education. They also urge researchers to examine how different 
aspects of these programs influence their effectiveness and to address the numerous barriers to 
the adoption of online instruction in higher education. 
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A lthough most student training 
and development in U.S. col¬ 
leges and universities continues 
to take place with teachers 
and students interacting face- 
to-face in traditional classrooms, the past 
decade has witnessed a significant migra¬ 
tion of postsecondary education from the 
classroom to online. A 2011 Babson Survey 
Research Group poll of more than 2,500 
chief academic officers found that 65 percent 
view online learning as a critical part of their 
long-term strategy. 1 The survey also revealed 
that more than 6 million, or 31 percent, of 
the nation’s college students took at least 
one online course during the fall 2010 term, 
an increase of more than 560,000, or 10.1 
percent, over the previous year. Although 
this figure is significantly lower than the 
21.1 percent annual growth in online enroll¬ 
ment recorded by Babson in fall 2009, it far 
exceeds the 0.6 percent annual growth in the 
overall number of higher education students 
during the same period. 

The growth of e-learning in postsecondary 
education is not limited to online courses 
and programs but rather covers an expanding 
array of applications and approaches that use 
technology in different ways and to varying 
degrees. These applications include simple 
videotaped lectures posted on the Internet, 
as well as learning-management systems, such 
as Blackboard, that distribute content such 
as lecture notes, syllabi, and assignments and 
facilitate peer and student-teacher interaction. 
They also include more sophisticated online 
collaborative simulations that create high- 
fidelity learning environments and interactive 
e-learning systems that use artificial intel¬ 
ligence to deliver customized instruction to 
students. 2 Interest is also growing in making 
learning accessible to students through mobile 
devices, such as smartphones and tablets. 


There are a variety of reasons for the growth 
of e-learning in postsecondary institutions, 
including a need to generate new revenue 
streams, expand access, offer students greater 
scheduling flexibility and the freedom to 
work at their own pace, and curb increas¬ 
ing costs. As e-learning has been expanding, 
however, so have debates about its effective¬ 
ness and concerns about its impact on the 
quality of higher education. According to 
Babson, for example, although two-thirds 
of the academic leaders polled believe that 
online education is just as good as or better 
than face-to-face instruction, the remaining 
one-third believe the learning outcomes of 
online courses are inferior to those of face- 
to-face instruction. 3 As might be expected, 
leaders at institutions that do not offer online 
courses or programs tend to be more skepti¬ 
cal. A survey of the general public conducted 
by the Pew Research Center using a nation¬ 
ally representative sample of 2,142 adults 
found that only 29 percent believe online 
courses are as valuable educationally as 
courses taken in the classroom. 4 

In this article we address three key questions 
about the growth of e-learning in postsecond¬ 
ary education. First, is e-learning as effective 
as other delivery media? The debate about 
the effectiveness of e-learning has typically 
been framed in terms of how it compares with 
other means of delivering instruction, particu¬ 
larly traditional teacher-led classroom instruc¬ 
tion. To examine this question we review 
research that evaluates the effectiveness of 
e-learning by comparing learning outcomes 
across different delivery media. Second, what 
features of e-learning influence its effective¬ 
ness? Exploring this issue requires moving 
beyond the “does it work” question to a more 
nuanced consideration of the conditions 
under which e-learning is likely to be most 
effective in postsecondary settings. Third and 
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finally, what are the barriers to the adoption 
of e-learning in higher education? Before 
addressing these questions, we define and 
describe e-learning and review current trends 
in how it is being used in higher education. 

What Is E-Learning? 

For researchers, e-learning is a vast and some¬ 
what disconnected area of inquiry that has 
attracted interest from disciplines as diverse 
as educational psychology, computer science, 
information science, management, communi¬ 
cations, and more. The breadth of the subject 
and the divergent objectives among those 
studying e-learning have led to a fragmented 
understanding of what e-learning means and 
how it should be defined. The current state of 
affairs is perhaps best illustrated by the many 
terms used to refer to instruction delivered 
through computer technology—e-learning, 
online learning, distance learning, distance 
education, computer-assisted instruction, 
computer-based instruction, technology-based 
instruction, technology-delivered instruction, 
computer-based simulation, and simulation 
games. In their recent review of e-learning 
research, Kenneth Brown, Steven Charlier, 
and Abigail Pierotti identified forty-six distinct 
terms. 5 One explanation for this prolifera¬ 
tion of terms is that the seemingly endless 
combinations and variants of technologies 
create different e-learning applications with 
very different capabilities. Another is the 
constantly evolving nature of e-learning, with 
new terms accompanying the introduction of 
new e-learning technologies or applications. 
Further complicating matters, e-learning can 
be used either as a stand-alone delivery tool 
or as a supplement to face-to-face instruction 
(the latter commonly known as “hybrid” or 
“blended” learning). As William Bowen and 
several colleagues suggest, “online learning’ 
is hardly one thing. It comes in a dizzying 
variety of flavors.” 6 


The challenge for those seeking to make 
sense of this field is that these terms are 
often applied inconsistently. For example, 
the terms “e-learning” and “online learning” 
are frequently used to refer to instruction 
in which most (often 80 percent or more) of 
the content is delivered through networked 
technology (such as the Internet), although 
the same terms have also been used to 
refer to programs delivered through non- 
networked digital technologies (for example, 
CD or DVD). Further, some analysts distin¬ 
guish between specific delivery media, such 
as “online learning,” and broad approaches 
to instruction, such as “distance education,” 
whereas others use these terms interchange¬ 
ably. Such inconsistent use of terms can make 
it difficult to determine the equivalency of 
courses or programs examined across studies. 
Accordingly, our view is that it matters less 
what specific label or term analysts use for a 
particular learning program than that they 
provide clear and detailed information about 
the technological and instructional features 
embedded in it. That is, it is important for 
investigators to describe the defining features 
of the e-learning programs they examine 
so findings across different studies can be 
appropriately aggregated and compared. In 
reality, however, authors commonly neglect 
to report important details about the learn¬ 
ing technologies and learner experiences 
they examine. 

We use the umbrella term “e-learning” to 
refer to all forms of electronically supported 
instruction. In their review, Brown, Charlier, 
and Pierotti formally define “e-learning” as 
“a broad array of applications and processes 
that share a common feature of relying on 
some type of computer technology to pro¬ 
mote learning.” 7 That expansive definition 
fits nicely with our own objective, which is 
to provide a broad review of technology in 
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postsecondary education. Another feature 
common to most postsecondary e-learning 
is that it relies on the Internet. 8 As noted, 
the term “e-learning,” along with “online 
learning” and “web-based instruction,” usu¬ 
ally refers to instruction delivered through 
network technology. Finally, we use the 
term “e-learning” because it is commonly 
used among the general public, as well as by 
colleges when they refer to their technology- 
enabled courses or degree programs. For 
example, a Google search yields about 
94 million results for “e-learning,” compared 
with 33 million for “distance learning” and 
20 million for “online learning.” Although 
we use this broad term throughout the 
article, we are careful to note when particu¬ 
lar findings or conclusions are confined to 
a specific type of technology. 

Current Trends in Postsecondary 
E-Leaming 

Institutions of higher education are putting 
much thought into how they might optimize 
their course enrollment and attract new 
students by delivering instruction through 
e-learning applications. Among the most 
active participants in the college e-learning 
market are large state systems such as the 
University of Maryland University College 
(UMUC), Pennsylvania State University, 
and the University of Massachusetts 
(UMassOnLine). In 2011, more than 70,000 
students worldwide enrolled in at least one 
online UMUC course and the university had 
more than 230,000 enrollments in online- 
only courses. UMUC, which has been educat¬ 
ing students in Europe since 1949 and in Asia 
since 1956, claims to be the largest four-year 
public university in the United States and the 
largest public provider of higher education 
to working professionals and to U.S. military 
personnel and their family members. 


For-profit universities are also a growing part 
of the online college market. The University 
of Phoenix (UP), which describes itself as the 
largest private university in North America, 
enrolled more than 380,000 students in 
degree programs in 2011. In its more than 
100 degree programs at the associate’s 
through the doctoral level, students can 
attend class online, in a traditional classroom, 
or a combination of both. Its 2011 Annual 
Academic Report noted that 2.2 percent 
of its students are nonresident aliens, 18.4 
percent are black, 36.3 percent are white, and 
68.9 percent are female. More than half of 
the graduate student body consists of minor¬ 
ity students. 9 As of September 2010, Kaplan 
University offered ninety-six academic 
programs, including fifty-nine degree pro¬ 
grams (associate’s, bachelor’s, and graduate), 
two diploma programs, thirty-two certificate 
programs, and three law-related degrees 
through distance, blended online, and on- 
campus learning. According to Kaplan’s 
2010 annual report, of the more than 68,000 
students enrolled during 2009—10, 75 percent 
were women and 55 percent were over the 
age of thirty. 10 Other prominent for-profit 
institutions include Laureate International 
Universities, which enrolls students from 
more than 120 countries in bachelor’s, 
master’s, and doctoral programs, and Strayer 
University, which offers associate’s, bachelor’s, 
and master’s degrees in a variety of areas, 
including business administration, account¬ 
ing, and information technology. The article 
by David Deming, Claudia Goldin, and 
Lawrence Katz in this issue examines for- 
profit colleges in detail. 11 

Over the past decade, the number of stu¬ 
dents enrolling in e-learning courses at these 
and other postsecondary institutions has 
grown dramatically. The National Center for 
Education Statistics estimates that between 
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2000 and 2008 the share of undergraduates 
enrolled in at least one online course grew 
from 8 percent to 20 percent. 12 As noted, the 
Babson Survey Research Group estimated 
that by the fall of 2010, 31 percent of all 
higher education students were taking at 
least one online course. 13 Further, Babson 
estimated that between 2002 and 2010 
online enrollments grew at a rate of 18.3 per¬ 
cent, compared with just more than 2 per¬ 
cent for the overall postsecondary education 
student body. A 2011 Pew Research Center 
survey of more than 1,000 U.S. colleges and 
universities found that 82 percent of commu¬ 
nity colleges offer online courses, compared 
with 79 percent of research universities and 
61 percent of liberal arts colleges. 14 It also 
found that 91 percent of two-year colleges 
offered online classes, compared with 
89 percent of four-year public colleges and 
universities and 60 percent of private col¬ 
leges and universities. 


The National Center for 
Education Statistics estimates 
that between 2000 and 2008 
the share of undergraduates 
enrolled in at least one online 
course grew from 8 percent 
to 20 percent. 


According to a 2011 survey conducted by the 
Instructional Technology Council, the share 
of students taking online classes at commu¬ 
nity colleges is split almost equally between 
traditional students aged eighteen through 
twenty-five (48 percent) and nontraditional 
students twenty-six and over (47 percent). 13 


Some 62 percent of online students are 
female; 37 percent, male. Although online 
courses and degrees are offered in a wide 
array of subject areas, the online-only 
bachelor s degree major that enrolls the most 
students is criminal justice. As estimated 
by the firm Eduventures, 27 percent of all 
online-only enrollments are in criminal jus¬ 
tice, followed by 19 percent in computer and 
information technology, 16 percent in health 
care, and 14 percent in business. 16 Online 
enrollments are estimated to be growing 
most rapidly in fields related to health care. 17 

Is E-Learning as Effective as 
Other Delivery Media? 

Even as online enrollment continues to grow, 
concerns remain about the legitimacy and 
value of e-learning in postsecondary educa¬ 
tion. The debate about the effectiveness 
of e-learning has historically been cast in 
terms of how electronic delivery of instruc¬ 
tion compares with other forms of delivery, 
particularly traditional classroom delivery, 
which remains the most common form of 
instruction in higher education. For rea¬ 
sons we discuss later, we do not find studies 
comparing the effectiveness of different 
media terribly enlightening. But because 
this comparison has attracted significant 
attention not only from academics but also 
from administrators and the general public, 
we next provide an overview of academic 
and public perspectives on the comparative 
effectiveness of e-learning and other delivery 
media. We then review empirical evidence 
on effectiveness and discuss its implications 
for e-learning in postsecondary settings. 

Academic and Public Perspectives 
on the Effectiveness of E-Leaming 

Among researchers, views on the relative 
effectiveness of e-learning and traditional 
instruction fall into two primary camps. 
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Richard Clark has argued that there is noth¬ 
ing uniquely advantageous to any delivery 
medium and that, therefore, a well-designed 
media comparison study should find no 
effects. 18 Clark sees technology as a mere tool 
to be manipulated at the hands of instruc¬ 
tional design, pedagogical approaches, and 
teacher practices. As he puts it, “media only 
deliver instruction but do not influence learn¬ 
ing.” 19 In the same vein, Steven Ross, Gary 
Morrison, and Deborah Lowther contend 
that “educational technology is not a homo¬ 
geneous ‘intervention’ but a broad variety of 
modalities, tools, and strategies for learning. 
Its effectiveness, therefore, depends on how 
well it helps teachers and students achieve 
the desired instructional goals.” 20 In sum, 
according to this view, e-learning should 
be no more or less effective than any other 
form of instructional delivery. As with other 
types of instruction, e-learning’s effectiveness 
depends on how well it is designed to cre¬ 
ate the instructional experience that makes 
learning possible. 

Academic advocates of e-learning, by con¬ 
trast, cite numerous potential pedagogical 
benefits, such as customizing instruction 
to the learner, creating multimedia (text, 
images, sound, video) information environ¬ 
ments, and increasing interactivity. 21 Recause 
many media tools today have moved beyond 
simple prerecorded videos and can now offer 
more interaction between learners and teach¬ 
ers, among learners, and between the learner 
and the content, some observers argue that 
different delivery media can offer unique 
learning support. They contend that in cer¬ 
tain situations e-learning can lead to better 
academic outcomes by creating an instruc¬ 
tional experience that is difficult or impos¬ 
sible to create in the classroom or through 
alternative media. 


A third perspective on the effectiveness of 
e-learning, more prevalent among the general 
public than among academics, is that the 
outcomes associated with e-learning courses 
are inferior to those of traditional, face-to- 
face instruction. William Rowen and his 
co-authors cite “concerns that at least some 
kinds of online learning are low quality and 
that online learning in general de-personal- 
izes education.” 22 As noted, the Pew Research 
Center reports that a majority of the gen¬ 
eral public, including young adults who 
have grown up in a digital world, believes 
that online courses offer less educational 
value than traditional classroom courses. 23 
Lawrence Racow and several colleagues also 
note that many faculty are skeptical of the 
value of e-learning because it differs from the 
way in which they were taught and because 
they fear it will distance them from their stu¬ 
dents, thereby undermining the educational 
and mentoring process. 24 

Empirical Evidence on 
Effectiveness 

Over the past several decades, thousands 
of studies have examined the effective¬ 
ness of e-learning, broadly defined. Much 
of this work has compared e-learning with 
traditional classroom instruction and other 
forms of delivery media. Early research 
focused primarily on evaluating distance 
education, such as televised broadcasts and 
videoconferencing, but over time attention 
shifted to computer-based instruction and 
most recently to online instruction as well as 
computer-based simulations. Proponents of 
this research argue that a systematic account 
explaining why and how learning effective¬ 
ness differs between different forms of deliv¬ 
ery could help policy makers, administrators, 
researchers, and educational-design special¬ 
ists determine the equivalency and value of 
ongoing innovation. 25 
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Over the years, however, e-learning research 
has been hamstrung by several important 
methodological limitations. Although these 
deficiencies are sometimes beyond the 
control of investigators, they have nonethe¬ 
less led to questions about the validity of 
the research findings. 26 For example, many 
studies have used what is called “single group 
pretest, post-test designs,” which can lead to 
an upward bias in effect sizes. 27 And even in 
more sophisticated two-group study designs 
that compare treatment groups with com¬ 
parison, or control, groups, participants are 
often not randomly assigned to treatment and 
control conditions (only in the gold standard, 
or experimental, study designs are partici¬ 
pants assigned randomly). Participants may 
thus self-select into different instructional 
conditions, which can allow preexisting dif¬ 
ferences among them to go unmeasured and 
lead to bias in observed effects. As a second 
illustration of methodological limits, some 
studies, even those with comparison groups, 
can confound differences in delivery media 
with differences in instruction. 28 In other 
words, the instruction received by partici¬ 
pants in the e-learning condition is often not 
equivalent to that received by participants in 
the classroom or comparison condition. In 
certain forms of e-learning, such as simula¬ 
tions, for example, students may be required 
to engage more actively than they would in a 
classroom environment. Because instructional 
methods that facilitate active engagement 
enhance learning, differences in achievement 
may be attributable to differences in activ¬ 
ity level rather than in the delivery media 
per se. 29 The curriculum materials and the 
time spent in learning can also differ across 
the e-learning and comparison groups, thus 
leading to differences in achievement. In 
short, because differences in delivery media 
are often associated with differences in other 
instructional features, studies that contrast 


different delivery media are often not making 
apples-to-apples comparisons. 

A comprehensive review of this vast and 
diverse research literature is beyond the 
scope of this article. Instead, we focus on 
several meta-analyses that have been con¬ 
ducted on e-learning research. Meta-analysis 
is a technique for combining the results of 
multiple studies to obtain an overall estimate 
of a particular effect or relationship. One of 
the advantages of meta-analysis is that by 
aggregating the findings of multiple studies it 
reduces the influence of factors that are idio¬ 
syncratic to a specific study. In addition, the 
meta-analyses that we examine acknowledge 
the methodological limitations noted above 
and take steps to try to address them. For 
example, all of the meta-analyses screened 
studies to ensure they met a basic level of 
methodological rigor, such as employing if 
not an experimental design, at least a quasi- 
experimental design, in which there is a 
comparison group that receives a comparable 
treatment and often an attempt to statisti¬ 
cally control for differences between the 
students engaged in e-learning and those 
engaged in other forms of learning. Studies 
that did not meet these minimum standards 
were excluded from the meta-analyses. 
Furthermore, each of the meta-analyses 
recorded methodological and substantive 
differences across studies, such as whether 
the curriculum and instruction was equiva¬ 
lent in the treatment and comparison condi¬ 
tions, and examined these differences to see 
whether they affected the results. Despite 
these efforts, the studies included in the 
meta-analyses vary significantly in terms of 
methodology. And because many studies 
provide limited information about the nature 
of instruction in different conditions, it is 
impossible to account fully for potentially 
important instructional differences that may 
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be influencing the findings. For example, in 
their meta-analysis, Robert Bernard and his 
co-authors note that the studies of e-learning 
they reviewed commonly describe thor¬ 
oughly the e-learning condition, but offer 
little detail about the comparison condition 
(classroom instruction). 30 

These limitations notwithstanding, we believe 
the meta-analyses provide the most compre¬ 
hensive assessment of the effectiveness of 
e-learning relative to other delivery media. 
We next review the findings of several meta¬ 
analyses, emphasizing more recent reviews 
because they often incorporate studies 
analyzed in earlier reviews and are also more 
likely to include studies that examine the 
effectiveness of modern forms of e-learning 
such as web-based learning and interactive 
simulations. In addition, we devote attention 
to reviews that focus primarily on adult learn¬ 
ers because those findings are more likely to 
generalize to college students. 

Results of Meta-Analysis 
The meta-analysis conducted by Bernard 
and his colleagues examined 232 studies 
(yielding 688 effect sizes) dated from 1985 to 
2002 that compared e-learning (which they 
termed distance education) with traditional, 
or classroom, instruction on measures of 
achievement, student attitude, and course 
completion. 31 The studies focused on two 
types of e-learning—asynchronous (mostly 
correspondence and online courses, in which 
students participate at different times) and 
synchronous (mostly teleconferencing and 
satellite-based delivery, in which all students 
participate simultaneously)—and included 
a mixed population of students, including 
K-12, graduate, and military, although most 
were undergraduates. Measures of student 
achievement showed no significant overall 
difference between e-learning and classroom 


instruction. Measures of student attitude 
showed a small but significant difference, 
with students generally favoring classroom 
instruction over e-learning, although they 
rated only synchronous e-learning sig¬ 
nificantly lower than they rated classroom 
instruction. Course completion measures 
showed a very small but significant overall 
difference in favor of classroom instruc¬ 
tion, though only when compared with 
asynchronous e-learning. In summary, the 
meta-analysis revealed no significant overall 
difference between e-learning and traditional 
instruction in terms of overall achievement, 
but more negative student attitudes toward 
synchronous e-learning and higher dropout 
rates in asynchronous e-learning. 


Measures of student attitude 
showed a small but significant 
difference, with students 
generally favoring classroom 
instruction over e-learning, 
although they rated only 
synchronous e-learning 
significantly lower than they 
rated classroom instruction. 


A meta-analysis by Traci Sitzmann and 
several colleagues compared the effective¬ 
ness of classroom and web-based instruction, 
defined as a “hypermedia-based instructional 
program, which utilizes the attributes and 
resources of the World Wide Web to create 
a meaningful learning environment where 
learning is fostered and supported.” 32 Their 


172 THE FUTURE OF CHILDREN 




E-Learning in Postsecondary Education 


analysis of ninety-six published and unpub¬ 
lished studies involving 19,331 students found 
that web-based instruction was 6 percent 
more effective than traditional classroom 
instruction for teaching declarative knowl¬ 
edge (facts and principles), but not procedural 
knowledge (rules and procedures) or student 
reactions. Used as a supplement to classroom 
instruction (blended learning), web-based 
instruction was 13 percent more effective 
than classroom instruction for declarative 
knowledge and 20 percent more effective 
for procedural knowledge. These findings, 
however, should be interpreted with caution 
because offering web-based instruction as a 
supplement may lead to more learning time 
or other important instructional differences 
relative to the comparison classroom condi¬ 
tion. Indeed, the authors found web-based 
and classroom instruction equally effective 
for teaching declarative knowledge when 
the instructional methods used in both were 
equivalent. They attribute the small overall 
advantage of web-based instruction to its use 
of more (and more effective) instructional 
methods, rather than to the delivery media 
per se. 

Another recent meta-analysis, conducted 
by the U.S. Department of Education, 

Office of Planning, Evaluation, and Policy 
Development, examined fifty independent 
effect sizes from forty-five studies evaluat¬ 
ing online learning. 33 Although the meta¬ 
analysis was designed to draw conclusions 
about online learning in the context of 
K-12 education, only five studies with K-12 
students met the inclusion criteria set by the 
authors. The remaining studies focused on 
college undergraduates or those in gradu¬ 
ate programs or professional training. The 
findings revealed that students who took a 
course online did not perform significantly 
differently than those taking the same course 


through traditional face-to-face instruction. 
Students in courses that combined online 
and face-to-face instruction (blended learn¬ 
ing) had stronger learning outcomes than 
did those in face-to-face instruction alone. 
Both instructor-directed and collaborative 
and interactive online instruction (both fully 
online and blended) led to stronger outcomes 
than classroom instruction, but outcomes in 
independent online learning and face-to-face 
instruction had no significant difference. 
Finally, the positive effect of online learning 
(both fully online and blended) was reduced 
somewhat when curriculum materials and 
instructional approach were equivalent 
across conditions. 

In a study published in 2011, Sitzmann used 
meta-analytic techniques to examine the 
instructional effectiveness of computer-based 
simulation games. 34 The studies that she 
analyzed used different kinds of comparison 
groups, with participants in some receiv¬ 
ing no training and those in others receiv¬ 
ing alternative instructional methods. To be 
included in the meta-analysis, a study had 
to focus on adult learners (aged eighteen or 
older) and on training that facilitated poten¬ 
tially job-relevant knowledge or skills. The 
analysis, which covered a total of sixty-five 
independent samples from fifty-five reports, 
revealed that trainees in the simulation game 
group had 11 percent higher declarative 
knowledge, 14 percent higher procedural 
knowledge, 9 percent higher retention, and 
20 percent higher self-efficacy than trainees 
in the comparison group. The entertainment 
value of the simulation did not influence its 
effectiveness, nor did differences in method¬ 
ology across studies (for example, studies with 
and without random assignment). Results 
did vary, however, by the type of instruction 
provided to the comparison group and the 
simulation group. Simulation games were 
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more effective than lectures, assignments, 
and readings, but less effective than comput¬ 
erized tutorials. Trainees learned more from 
simulation games when they had unlimited 
access to the games (presumably leading 
to more time spent learning) and when 
the games were embedded in a program of 
instruction (blended learning). In fact, when 
simulation games were the sole instructional 
method, trainees in the comparison group 
learned more than those in the simulation 
game group. Finally, in studies that matched 
the simulation and comparison groups in 
terms of the activity level of instruction, 
learning was similar across conditions. Once 
again, this finding suggests that the learners 
in the simulation games condition may have 
been advantaged not because of the delivery 
media per se, but rather because they often 
received more active instruction than those in 
the comparison group. 

Effectiveness of E-Learning: Conclusions 
These meta-analytic studies paint a rather 
complex picture of the effectiveness of 
e-learning. Overall their findings, as well as 
the findings of earlier reviews not discussed 
here, suggest that e-learning is at least as 
effective as, and in some cases more effec¬ 
tive than, classroom instruction. But taking 
into account various methodological and 
instructional factors can change the find¬ 
ings—typically not reversing them but rather 
weakening or eliminating the observed ben¬ 
efits of e-learning. Furthermore, some of the 
meta-analyses found widely varying effect 
sizes for the relationship between e-learning 
and the learning outcomes, with some stud¬ 
ies finding e-learning much more effective 
than classroom instruction and others find¬ 
ing it much less effective. Such variability 
suggests that other explanations—such 
as aspects of the instruction, teacher effec¬ 
tiveness, or student characteristics—account 


for the relative effectiveness of e-learning in 
the studies. 

Several recent studies that have attempted 
to address the deficiencies of earlier work 
in this area have provided a more rigor¬ 
ous evaluation of the effect of e-learning on 
student achievement. David Figlio, Mark 
Rush, and Lu Yin, for example, randomly 
assigned students in a large introductory 
microeconomics course to either live lectures 
or online delivery of the recorded lectures. 35 
The sections differed only in the method 
of delivery and were identical in all other 
ways, including the instruction, assignments, 
and teaching assistant support. Overall, the 
course exam scores for students in the live 
instruction and online sections showed no 
significant difference, although certain stu¬ 
dents—specifically, Hispanic students, males, 
and low achievers—performed significantly 
better in the live instruction section. In a 
study already noted, William Bowen and his 
co-authors randomly assigned students in an 
introductory statistics course conducted at 
six public universities to either a traditional 
classroom-based section or a hybrid section. 36 
In the hybrid section, most of the instruc¬ 
tion was delivered through interactive online 
materials, but students also attended for one 
hour a week a face-to-face session where they 
could ask questions and receive assistance. 
Student learning outcomes in the traditional 
and hybrid sections showed no statistically 
significant difference. Nor did outcomes 
differ across subgroups—whether by race 
and ethnicity, gender, or college grade point 
average—indicating that no subgroups of 
students consistently benefited from or were 
harmed by the hybrid format. The rigorous 
design of both of these studies made pos¬ 
sible a precise estimate of the differences (or 
lack thereof) between conditions. As a result, 
though both reached the same conclusion 
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as the meta-analyses—that e-learning is, on 
average, as effective as traditional classroom 
instruction—their use of random assignment 
and strong controls engenders more confi¬ 
dence in their findings. 

So, what can we ultimately conclude from the 
multitude of studies comparing the effective¬ 
ness of e-learning and other forms of instruc¬ 
tion? The current body of evidence appears 
to support the position asserted by Richard 
Clark more than two decades ago: pedagogy, 
not delivery media, is what influences learn¬ 
ing. Or as Terry Anderson observes, “It seems 
clear that there is no single medium that sup¬ 
ports the educational experience in a manner 
that is superior in all ways to that supported 
via other media.” 37 Rather, characteristics of 
the instructional design, such as the instruc¬ 
tional methods used, the feedback provided, 
and the degree of learner engagement, create 
the conditions within which learning occurs. 
The meta-analyses reviewed above show that 
when instructional design characteristics 
are held constant across delivery conditions, 
e-learning and classroom instruction gener¬ 
ally produce similar learning outcomes. That 
finding suggests that delivery media them¬ 
selves do not affect learning, but rather are 
simply the vehicles through which instruc¬ 
tional conditions are delivered to the learner. 
Furthermore, the finding suggests that 
studies designed to evaluate the effectiveness 
of a particular e-learning technology are of 
limited value. Indeed, any form of instruction 
can be effective if it is able to create the con¬ 
ditions necessary for students to learn specific 
content. As Brown, Charlier, and Pierotti 
conclude in their review of e-learning, “we 
could study whether people learn using iPods 
with no screen, iPods with a small screen, 
e-readers (e.g., Nooks, Kindles) with black- 
and-white or color screens, iPads (which have 
a larger color screen), and cinsert future 


technology hero. Invariably, the answer to 
the question will be, yes, people can learn 
using these media” (italics in original). 38 
Similarly, studies that simply compare differ¬ 
ent media, without considering differences 
in instructional methodology or learning 
environments, do not provide an accurate pic¬ 
ture of the effectiveness of one type of media 
relative to another. Ultimately, research needs 
to move beyond the “does it work” question 
toward a better understanding of exactly what 
does influence the effectiveness of e-learning 
and thus of the conditions under which 
e-learning is likely to be most effective. 

What Features of E-Learning 
Influence Its Effectiveness? 

Researchers have now begun to investigate 
the effectiveness of e-learning by evaluating 
not the different technologies themselves but 
rather the effects of specific instructional 
features and supports embedded in them. 
Several authors have developed conceptual 
frameworks or typologies of e-learning to 
help guide such efforts. Steve Kozlowski 
and Bradford Bell, for example, present a 
typology that highlights four key categories 
of instructional features—content, immer¬ 
sion, interactivity, and communication—by 
which e-learning technologies can create a 
specific instructional experience. 39 By content 
they mean the level of richness with which 
information is delivered to learners. Text, 
for example, is low in information richness, 
whereas images, sound, and video are high. 
Kozlowski and Bell use the term immersion 
to denote the sense of realism that e-learning 
can create—the extent to which the learning 
experience captures the psychological and 
physical characteristics of a performance. 
Certain forms of technology, for example, 
such as simulations, offer greater possibili¬ 
ties for enhancing learners’ sense of being 
immersed in the educational experience. 
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The term interactivity refers to character¬ 
istics that influence the degree and type of 
interaction between learners (individually 
or as groups), between learners and instruc¬ 
tors, and, increasingly, between learners 
and simulated characters or virtual agents. 
Networked forms of e-learning, for example, 
have expanded the potential for collaboration 
and interactivity. The final feature, commu¬ 
nication richness or bandwidth, determines 
students’ ability to communicate verbally 
and nonverbally. E-learning programs dif¬ 
fer in the extent to which they make avail¬ 
able different communication channels (for 
example, audio and video) and allow students 
to communicate synchronously in real time. 
Kozlowski and Bell stress that the importance 
of different features depends on the goals 
of a particular learning program and the 
instructional environment that must be cre¬ 
ated to meet the needs oflearners. That is, no 
configuration of features will be universally 
effective or ineffective. Rather, effectiveness 
is determined by the degree of fit between 
the design of the e-learning and the char¬ 
acteristics of the course for which it is used. 
Educators can use research of this sort—work 
that focuses on how different technological 
configurations can deliver specific instruc¬ 
tional features—to guide decisions about 
which type of e-learning should be used to 
meet specific learning objectives. 

Other conceptual work has focused on 
isolating specific instructional features of 
e-learning. Interactivity has received par¬ 
ticular attention. In a review of computer 
games and simulations, Jennifer Vogel and 
several colleagues argue that interactivity is 
the key instructional component that influ¬ 
ences learning outcomes. 40 In 1989, Michael 
Moore identified three forms of interaction 
in distance education: student-student inter¬ 
action, student-instructor interaction, and 


student-content interaction. 41 Drawing on 
Moore’s typology, Terry Anderson proposed 
that e-learning can support meaningful 
learning as long as at least one form of inter¬ 
action is at a high level. 42 High levels of mul¬ 
tiple forms of interaction may enhance the 
educational experience, but may make it less 
cost- or time-effective. What Anderson calls 
his “equivalency theorem”—that one type 
of interactivity can substitute for the others 
with little loss in educational effectiveness— 
further delineates the different forms of 
interactivity and shows how different tech¬ 
nologies can meet learner needs through dif¬ 
ferent types of interactivity. Future research 
must directly test the equivalency theorem 
to learn whether specific types of interaction 
are better suited than others to meet specific 
learner needs and instructional objectives. 

Empirical research is also shifting away from 
evaluating whether e-learning works and 
toward examining the instructional features 
that influence its effectiveness. Rather than 
comparing different forms of delivery such 
as e-learning versus classroom, studies are 
beginning to compare e-learning programs 
that differ on important instructional dimen¬ 
sions, including interactivity, engagement 
and activity, and feedback. Richard Mayer, 
for example, has conducted research on 
multimedia learning to better understand 
how people learn in such environments and 
to identify which aspects of those environ¬ 
ments can help different types of learners 
acquire different kinds of knowledge. 43 
Robert Bernard and several coauthors have 
conducted a meta-analysis to examine how 
different types of interaction influence the 
effectiveness of e-learning programs, which 
they call distance education. 44 Based on 
seventy-four effect sizes drawn from seventy- 
four studies, they found that programs 
offering moderate to high levels of interaction 
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had better achievement outcomes than those 
offering less interaction. They found, interest¬ 
ingly, that programs that incorporated stu¬ 
dent-student or student-content interaction 
led to better achievement than those offering 
student-teacher interaction. They also found 
that increasing the degree of interaction led 
to better achievement in the case of student- 
content interaction, but not student-student 
or student-instructor interaction. To repeat, 
the effects of a certain type of interaction 
will depend on how well it matches the 
content and objectives of a particular course. 
Yet, as this meta-analysis shows, on aver¬ 
age, interactivity significantly influences the 
effectiveness of e-learning programs, and 
certain types of interaction may lead to bet¬ 
ter outcomes than others. 

In summary, research provides evidence that 
e-learning can effectively deliver instruction 
in postsecondary settings. As with any deliv¬ 
ery media, whether a particular e-learning 
program is effective in a given situation will 
depend on its capacity to create the condi¬ 
tions necessary for students to learn. The key 
challenges now facing college administra¬ 
tors and faculty are to decide when to use 
e-learning and how to design and deliver 
it to maximize student achievement. As 
yet, however, e-learning research provides 
minimal guidance on these central questions. 
In other areas, such as the organizational 
training literature, researchers have mapped 
the effectiveness of specific training design 
features, such as lecture, self-instruction, or 
discussion, as a function of the skill or task 
being taught. 45 Such research can guide deci¬ 
sions about what methods should be used 
to teach different skills or tasks. As noted, 
similar research evaluating the effectiveness 
of e-learning features such as interactivity 
and immersion for teaching different con¬ 
tent would help curriculum planners decide 


when e-learning is appropriate and what type 
of e-learning should be used to deliver the 
features critical to learning in a particular 
course or program. 

Barriers to E-Learning 
in Postsecondary Education 

Observers have pointed to a number of 
potential obstacles to e-learning instruction 
in higher education. 46 In this final section we 
examine several emerging issues and trends 
that we believe may create significant barriers 
to the widespread adoption of e-learning in 
the nation’s colleges and universities. 

Fraud and Cheating Online 

As evidence accumulates about how to make 
online learning effective, concerns are grow¬ 
ing about problems that e-learning poses 
for students’ academic integrity. Academic 
dishonesty has typically been characterized 
by the following offenses: “acts of plagia¬ 
rism, using concealed notes to cheat on tests, 
exchanging work with other students, buy¬ 
ing essays or, in some extreme and notorious 
cases, asking others to sit examinations for 
you.” 47 Research has long documented the 
widespread prevalence of such forms of dis¬ 
honesty in postsecondary institutions. 

In 1964, for example, Bill Bowers published 
the first large-scale study of self-reported 
cheating in postsecondary institutions. 48 In 
a sample of more than 5,000 students from 
ninety-nine U.S. colleges and universities, he 
found that three-quarters of the students had 
engaged in at least one dishonest academic 
behavior. During the 1993-94 academic 
year, Donald McCabe and Linda Trevino 
surveyed approximately 1,800 students at 
nine of the schools that had participated in 
Bowers’s original study. 49 They found that 
although the share of students who cheated 
had increased only slightly, from 63 percent 
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in 1963 to 70 percent in 1993, cheaters from 
the 1993 group engaged in a wider variety of 
cheating, cheated more often, and engaged 
in more forms of exam cheating. The share of 
students admitting to collaborating on indi¬ 
vidual assignments jumped from 11 percent 
in 1963 to 49 percent in 1993. 

More recently, studies have begun specifi¬ 
cally to examine academic dishonesty in 
online learning environments. In 2006, 

Mark Lanier surveyed 1,262 students at a 
large, state-funded university and found 
that self-reported cheating was more preva¬ 
lent in online classes than in traditional 
lecture courses. 50 In 2000 Kristen Kennedy 
and several colleagues found that both 
students and administrators believe it is 
easier to cheat in distance learning classes. 51 
Kenneth Chapman and several colleagues 
conducted a survey of 824 business stu¬ 
dents, both undergraduate and graduate, 
and found that approximately 75 percent 
admitted to cheating at some point in their 
courses. 52 Among those who had taken an 
e-learning course, 24 percent admitted to 
having cheated on a web-based examina¬ 
tion. More strikingly, 42 percent indicated 
that they would cheat on electronic exams 
if given the opportunity. A recent report 
by the Government Accountability Office 
(GAO) also reported a set of alarming find¬ 
ings pertaining to faculty and administrative 
toleration of academic dishonesty in online 
courses at for-profit institutions. 53 

Many institutions are exploring ways to 
address cheating in online courses, includ¬ 
ing having students take exams on campus 
or in testing centers or replacing high-stakes 
testing with assessments, such as longer-term 
projects, that are seen as less susceptible to 
cheating. 54 Perhaps more important, col¬ 
leges must ensure that institutional policies 


regarding academic dishonesty and course 
grading standards are followed strictly to 
create a culture of academic integrity in the 
online environment. The work of Donald 
McCabe, Linda Trevino, and Kenneth 
Butterfield suggests that perceived social 
norms, attitudes toward cheating, and knowl¬ 
edge of institutional policy regarding cheating 
will generally predict course conduct from 
students in online learning environments. 55 

Low-Income and Underprepared 
Students 

One argument in favor of e-learning is its 
potential to improve access to higher educa¬ 
tion among lower-income and academically 
underprepared students. Online learning, 
supporters say, makes postsecondary educa¬ 
tion more affordable, expands geographic 
access (for example, to rural areas), and 
provides needed flexibility for students who 
cannot attend traditional classes because of 
full-time work and child-care responsibili¬ 
ties. Realizing that potential, however, will 
not be easy. 

Over the past two decades, much public 
discussion has focused on “digital divides” 
and their implications for both youth and 
adults. In an article in the Encyclopedia 
of Adolescence Linda Jackson describes 
three generations of such divides, all by 
income and race. 56 The first generation was 
the divide in access to digital technologies, 
especially the Internet; access increased with 
income and was higher among whites than 
African Americans. As public access to the 
Internet increased in schools, libraries, and 
other public spaces, a second digital divide 
emerged, again primarily by income and race, 
this one based on broadband Internet access. 
Researchers, educators, and policy makers 
have argued that broadband access funda¬ 
mentally changes the way people interact 
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with the Internet, including how often they 
go online, how much time they spend, and 
what they do. As these gaps have shown signs 
of narrowing, yet a third digital divide, this 
one in the intensity and nature of Internet 
use, has gained attention. Research has 
shown that among those with access to the 
Internet, African Americans go online less 
often than their white peers. A gap is also 
growing between youth who use the Internet 
in diverse and engaging ways, such as social 
networking or searching for information 
about major life issues (health care, finding a 
job), and youth who use it as a more narrow 
and less engaging resource, such as seeking 
entertainment online through music or video 
clips. These types of divides too tend to be 
structured along racial and ethnic and socio¬ 
economic backgrounds. 

The digital divides, particularly the third- 
generation divide, can lead to differences not 
only in users’ cognitive, social, and psycholog¬ 
ical development but also in their technology 
skills and confidence. And because lack of 
confidence is one of the most frequently cited 
barriers to adult Internet use, these digital 
divides may, ironically, decrease enrollment 
in e-learning among the very groups for 
whom e-learning is supposed to expand post¬ 
secondary access. The divides may also raise 
dropout rates among students who enroll 
in e-learning. Online courses, in fact, often 
have significantly higher dropout rates than 
face-to-face courses. 5 ' One primary reason 
students give for dropping out is technical 
problems—problems that students without 
access to broadband Internet may be espe¬ 
cially likely to experience. 58 And students who 
lack technology skills and confidence may 
be less likely to persist when such problems 
arise. Thus, if e-learning is to increase access 
to college among low-income students and 
specific racial and ethnic groups, institutions 


will have to address digital divides in terms 
not only of students’ access to technology but 
also of their technology skills and literacy. 


Online courses, in fact, often 
have significantly higher 
dropout rates tha n face-to- 
face courses. One primary 
reason students give for 
dropping out is technical 
problems—problems that 
students without access to 
broadband Internet may be 
especially likely to experience. 


Underprepared students too may face barri¬ 
ers to success in e-learning courses. Figlio, 
Rush, and Yin, for example, found, in the 
study already noted, that students with low 
grade-point averages who enrolled in the 
e-learning section of a microeconomics class 
scored significantly lower on course exams 
than did those in the live instruction sec¬ 
tion. 59 Research examining underprepared 
students, though limited, has typically 
reached a similar conclusion: academically 
underprepared students often perform worse 
than their peers in online courses. The find¬ 
ing is not surprising in light of the importance 
of self-regulatory skills in learning generally. 
And given that e-learning often shifts to the 
learner more control over important learn¬ 
ing decisions, such as what and how much 
to study, self-regulatory skills such as self- 
monitoring and self-evaluation become even 
stronger predictors of student motivation, 
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achievement, and course completion. One 
way to improve the skills of underprepared 
students is to build instructional supports 
into e-learning courses to help students self- 
regulate and make effective use of the control 
they are given over their learning. 60 Several 
interventions that prompt self-regulation 
through reflective questions also show prom¬ 
ise for supporting critical learning processes 
during e-learning, though more work is 
needed to evaluate the specific effectiveness 
of these interventions among academically 
underprepared students. 61 Educators’ increas¬ 
ing interest in learning analytics also holds 
promise for overcoming the barriers faced 
by these students. Data collected from large 
populations of online learners can provide 
insight into the usage and performance of 
different types of learners and help curricu¬ 
lum planners design courses that meet the 
specific needs of underprepared students. 62 

Cost Issues 

Although one of the most common reasons 
given by academic leaders at postsecondary 
institutions for developing online courses and 
programs is to generate new revenue streams 
and potentially to lower the costs of providing 
a postsecondary education, the cost-effective¬ 
ness of e-learning remains largely an open 
question. Lawrence Bacow and his coauthors 
report that relatively few institutions believe 
e-learning reduces their costs, and, in fact, 
most believe that online courses are at least as 
expensive to provide as traditional courses. 63 
This perspective is based largely on the sig¬ 
nificant start-up costs of e-learning, including 
investments in technology, course design, 
and the training of instructors, but also on 
recurring costs, such as those that result from 
increased coordination demands and techni¬ 
cal support. These costs can be a significant 
barrier to entry for institutions seeking to 
adopt e-learning instruction. Indeed, Babson 


Survey Besearch Group estimates that a small 
subset of postsecondary institutions currently 
educate the majority of online students and 
predicts that future growth will come largely 
from those same institutions rather than from 
new institutions. 64 

Nevertheless, some analysts believe that 
e-learning can reduce the cost of education. 
Bowen and his coauthors, for example, ran 
several cost simulations to try to estimate 
how much the institutions in their study 
could save by shifting to hybrid learning. 65 
Although the simulations are speculative and 
the results vary depending on the assump¬ 
tions that are adopted, they show that hybrid 
learning may promise significant savings 
in total instructor compensation costs. In 
higher education today, far more e-learning 
courses are led by instructors rather than by 
machines, thus limiting the extent to which 
institutions can realize these cost savings. But 
future adoption of more interactive, machine- 
guided courses could significantly lower 
costs. 66 Tamara Battaglino, Matt Halderman, 
and Eleanor Laurans stress that the impor¬ 
tant question is not simply whether e-learning 
is cheaper but whether it can achieve similar 
or better learning outcomes at a lower cost. 67 

Other Unanswered Questions 
Several other important questions will 
require the attention of educators, adminis¬ 
trators, and policy makers as postsecondary 
e-learning continues to expand. One such 
question concerns the impact of e-learning 
on more distal measures of student achieve¬ 
ment, such as retention and the transfer of 
learning, both to other courses and to the 
workplace. Most e-learning studies assess 
student achievement during the course itself 
or immediately upon completion. When these 
studies assess retention at all, they usually 
do so within a month of when students finish 
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the course. 68 In addition, they often assess 
achievement based on students’ cognitive 
knowledge and attitudes, at the expense of 
other important learning outcomes, such 
as how they apply what they are learning. 
Admittedly, the failure to assess achievement 
using longer-term measures is not confined 
to research on e-learning; much of the adult 
learning research focuses on shorter-term, 
cognitive outcomes. Yet it is not possible to 
grasp fully the implications of e-learning in 
postsecondary settings without examining 
how it affects students’ ability to retain and 
apply what they have learned. 

As colleges increasingly seek to make their 
e-learning courses available to an interna¬ 
tional audience, it will also be important to 
conduct research that spans different coun¬ 
tries and cultures. Much of the research in 
this area has been conducted using college 
students in the United States, which raises 
questions about whether findings will trans¬ 
late across national and cultural boundaries. 
For instance, students’ attitudes toward and 
acceptance of e-learning may vary depending 
on their cultural norms, beliefs, and values. 

In addition, research has found that cultural 
differences in technology use and digital 
divides that have been largely addressed 
in the United States continue to persist 
in other parts of the world, such as rural 
China. 69 In other parts of the world, it may 
also be important to shift attention from how 
e-learning compares to classroom learning 
to how e-learning can provide postsecondary 
educational opportunities that otherwise do 
not exist. 


Conclusion 

The use of e-learning in postsecondary 
education has expanded rapidly over the past 
decade, and all indicators suggest that growth 
will continue in the years to come. E-learning 
has also attracted intensive research inter¬ 
est, with thousands of studies over the past 
several decades examining its effectiveness. 
Although the dominant paradigm in this 
area—comparing e-learning with classroom 
instruction—has long been faulted, research 
is only now beginning to move away from 
the “does it work” question toward a greater 
focus on understanding the role of different 
instructional features and supports in deter¬ 
mining the effectiveness of e-learning. Future 
research should use rigorous experimental 
designs to examine how e-learning programs 
that vary in terms of content, interactivity, 
and other important instructional features 
affect students’ ability to acquire different 
types of knowledge and skills. Yet advances in 
e-learning design must also be coupled with 
efforts to eliminate current barriers to the 
widespread adoption of online instruction. 
Academics and institutions need to collabo¬ 
rate to address the challenges surrounding 
academic integrity in online environments, 
devise effective support systems for under¬ 
prepared learners, evaluate the economic 
models that underlie e-learning, and under¬ 
stand how to deliver e-learning across geo¬ 
graphic and cultural boundaries. 
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